Skip to content

feat: export the telemetry of divan benchmarks over OTLP - #197

Open
lvaroqui wants to merge 6 commits into
mainfrom
cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat
Open

lvaroqui wants to merge 6 commits into
mainfrom
cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat

Conversation

@lvaroqui

@lvaroqui lvaroqui commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Export the traces and logs that divan benchmarks emit with tracing and log, for the telemetry mode of the CodSpeed runner (CodSpeedHQ/codspeed#589) to collect.

Using it takes the telemetry feature of codspeed-divan-compat and a telemetry build:

cargo codspeed build -m telemetry            # or with other modes, e.g. -m simulation,telemetry
codspeed run -m telemetry -- cargo codspeed run

When CODSPEED_OTEL_ENDPOINT is set, divan::main() installs a tracing subscriber exporting over OTLP/HTTP:

  • spans as traces;
  • tracing events and log records as logs, carrying the trace and span IDs of the span they are emitted in.

CODSPEED_TELEMETRY_FILTER takes EnvFilter directives, info by default. Benchmarks that install their own subscriber add divan::telemetry::layer() to it.

cargo-codspeed

  • A new telemetry measurement mode builds the benchmarks to run once, like simulation, with tracing kept. It can be built along other modes.
  • Every other mode compiles tracing out (max_level_off, behind --cfg=codspeed_tracing_off), so spans and events never weigh on the measurements. --keep-tracing opts out, e.g. to benchmark a subscriber.
  • The OpenTelemetry dependencies sit in a cfg(codspeed_telemetry) target table, a cfg only the telemetry build sets. With the feature on, measured builds compile 31 crates of codspeed instead of 157, and a project without the feature gets nothing new in its lockfile.

Timing

Spans and log records keep the wall clock times of the SDK. With instrument hooks, they also carry their times on the monotonic clock of the benchmark markers, which wall clock adjustments don't move: spans as codspeed.monotonic_start_ns and codspeed.monotonic_end_ns, log records as codspeed.monotonic_time_ns.

Worth a careful look

  • How the span times are taken, in crates/codspeed/src/telemetry.rs. tracing-opentelemetry takes a span's start when the tracing span is created, but only starts the OpenTelemetry span when it is first entered, e.g. on the first poll of an instrumented future. A layer after it starts the span on creation, through get_otel_context, to set the start then. Its end is taken when the span is last exited, and the SDK only appends attributes, so a layer before it records the last exit and sets the end once, when the span closes.
  • The filters hint TRACE as their max level, so that LogTracer keeps the level of the EnvFilter: without it, filter_fn lifted it to TRACE.
  • init warms the subscriber up with a span and an event that are not exported: the first span of the process took 18 to 31 µs longer than the next ones.
  • The exporter's own HTTP stack (hyper, reqwest, opentelemetry_*, …) is always filtered out, or exporting would emit more spans in an endless loop.
  • reqwest pulls icu_* 2.x crates, which require rustc 1.88: they are resolved with the MSRV-aware fallback resolver. sharded-slab fails to build with the lazy_static 1.0 that minimal-versions picks, hence a lazy_static = "1.1" floor.

crates/divan_compat/examples/benches/telemetry.rs emits telemetry in each of these ways: nested spans with events, an event outside of any span, records of the log crate, and nothing at all. The telemetry tests only build with --cfg codspeed_telemetry; CI runs them, and checks the MSRV and minimal versions, with it.

Closes COD-3862

…try feature

Add an opt-in `telemetry` feature to `codspeed` and `codspeed-divan-compat`. When
`CODSPEED_OTEL_ENDPOINT` is set, `divan::main()` installs a global `tracing`
subscriber that exports the spans and events emitted by benchmarks over
OTLP/HTTP, filtered by `CODSPEED_TELEMETRY_FILTER` (defaults to `info`), and
flushes them when the run ends. Spans from the exporter's own HTTP stack are
always filtered out to avoid an export loop.

Benchmarks that already set a global subscriber are left untouched; they can
compose `divan::telemetry::layer()` into their own subscriber instead.

The telemetry dependencies stay within MSRV and minimal versions. `reqwest`
pulls `idna_adapter` 1.2.2 and the `icu_*` 2.x crates, which require rustc
1.88: they are resolved with the MSRV-aware fallback resolver so the lockfile
builds on rustc 1.85. `sharded-slab` declares `lazy_static = "1"` but fails to
build with 1.0.x, which breaks the minimal-versions check: an optional direct
`lazy_static = "1.1"` dependency, enabled by the `telemetry` feature, raises
the floor.

Closes COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codspeed

codspeed Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 3.78%

3 warnings, 1 note

⚠️ Unable to measure the execution of 59 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 17 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 57 benchmarks have no flamegraph

  • The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze. (9 benchmarks)
  • The flamegraph wasn't generated: no execution data was recorded while the benchmark ran. Make sure the compiler didn't optimize the benchmark out. (48 benchmarks)

See affected benchmarks

⚡ 16 improved benchmarks
❌ 23 regressed benchmarks
✅ 567 untouched benchmarks
🆕 12 new benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime iter_batched_ref_large_input 6 ns 11 ns -45.45%
❌ Walltime iter_batched_large_input 9 ns 13 ns -30.77%
❌ CPU Simulation add_two_integers[(42, 13)] 288 ns 401.6 ns -28.29%
❌ CPU Simulation add_two_integers[(255, 255)] 288 ns 401.6 ns -28.29%
❌ Walltime graph_coloring[5] 1.1 µs 1.5 µs -26.85%
❌ CPU Simulation count_set_bits[0] 170.3 ns 225.9 ns -24.59%
❌ CPU Simulation add_two_integers[(1024, -1024)] 349.4 ns 463.3 ns -24.58%
❌ CPU Simulation count_set_bits[1024] 171.6 ns 227.1 ns -24.46%
❌ CPU Simulation count_set_bits[42] 173.5 ns 229 ns -24.26%
❌ CPU Simulation add_two_integers[(65535, -65535)] 361.1 ns 475 ns -23.98%
❌ CPU Simulation count_set_bits[255] 178 ns 233.5 ns -23.79%
❌ CPU Simulation count_set_bits[65535] 185.1 ns 240.7 ns -23.08%
❌ CPU Simulation recursive_memoized[BTreeMap<u64, u64>, 0] 771.4 ns 936.3 ns -17.62%
❌ Walltime permutations[6] 68.7 µs 78 µs -12.02%
❌ Walltime hamiltonian_cycle[5] 848 ns 950 ns -10.74%
❌ CPU Simulation a 568.5 ns 624.2 ns -8.93%
❌ CPU Simulation a 568.5 ns 624.2 ns -8.93%
❌ CPU Simulation iter_manual_simple 839.4 ns 894.9 ns -6.21%
❌ Walltime recursive_memoized[HashMap<u64, u64>, 0] 18 ns 19 ns -5.26%
❌ Walltime zeroed_buffer[1] 47.9 µs 50.3 µs -4.84%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat (1e3f5f8) with main (5caa8dc)1

Open in CodSpeed

Footnotes

  1. No successful run was found on feat/time-benchmarks-with-round-markers (5b54240) during the generation of this report, so main (5caa8dc) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

lvaroqui and others added 3 commits October 16, 2026 12:22
Events were only recorded as span events, so a receiver got no logs signal
and events emitted outside any span were lost. Export them as log records
to `/v1/logs` instead, through the SDK batch log processor, carrying the
trace and span IDs of the span they are emitted in. Spans no longer carry
the events, which would otherwise be exported twice.

The log bridge sets the event name of every record to the name of its
`tracing` callsite, which `tracing` generates as `event <file>:<line>` for an
event given no `name:`, and `log event` for a record of the `log` crate. In
the OpenTelemetry logs data model a record with an event name is an event, so
a log processor clears these generated names: only the events named at their
callsite keep one. It compares the name with the record's own location,
leaving any other name as is. The bridge's `experimental_metadata_attributes`
feature is enabled for that location: the `code.filepath`, `code.lineno` and
`code.namespace` attributes, which keep it exported once the name is cleared.

Refs COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spans and log records carry wall clock times, which cannot be lined up
exactly with the benchmark markers taken on the monotonic clock: the
wall clock can be adjusted while the benchmarks run.

With instrument hooks, each span carries the times it started and ended
at on the clock of the markers as its `codspeed.monotonic_start_ns` and
`codspeed.monotonic_end_ns` attributes, and each log record the time it
was emitted at as `codspeed.monotonic_time_ns`. Their timestamps stay the wall clock times
of the SDK, as standard tools expect.

The OpenTelemetry layer takes the start time of a span when the
`tracing` span is created, but only starts the OpenTelemetry span when it
is first entered or closed, e.g., on the first poll of an instrumented
future. A layer placed after it starts the span on creation, through
`get_otel_context`, to set the start then. The OpenTelemetry layer takes
the end time when the span is last exited, or closed if it never was, and
ends the span when it is closed. Since the SDK only adds attributes, a
layer placed before it records the time of the last exit, and sets the
end once, when the span is closed.

The SDK batch processors are now built with a queue of 65536 rather than
the default 2048, so that a benchmark emitting on every iteration doesn't
overflow it.

Refs COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elemetry mode

`tracing` spans and events weigh on the measurements of the code they
instrument. Every build for the simulation, walltime and memory modes
now passes `--cfg=codspeed_tracing_off`, behind which the `codspeed`
crate depends on `tracing` with `max_level_off`: features unification
compiles every span and event of the build out, dependencies included.
Unlike `--features tracing/max_level_off`, it doesn't require each
selected package to depend on `tracing` directly. `--keep-tracing` opts
out, e.g. to benchmark a `tracing` subscriber.

The new `telemetry` measurement mode builds the benchmarks to collect the
telemetry they emit instead of measuring them: they run once like in
simulation mode (`--cfg=codspeed`), keep `tracing`, and land in
`target/codspeed/telemetry`. It builds along other modes, each with its
own flags, e.g. `-m walltime,telemetry`. `--keep-tracing` is rejected
when telemetry is the only mode, since it implies it.

The telemetry build also passes `--cfg=codspeed_telemetry`. The
dependencies exporting the telemetry move to a target table on that cfg,
so the other builds don't compile them, even with the `telemetry`
feature: a benchmark enabling it builds 31 crates of `codspeed` instead
of 157 in measured builds. They only appear in the lockfile of the
projects enabling the feature, as before. Without the cfg,
`telemetry::init` and `telemetry::layer` return `None`, which keeps the
API of the feature. CI checks the MSRV and minimal versions, and runs
the telemetry tests, with the cfg.

`release_max_level_*` features still take precedence over
`max_level_off` in measured builds, since debug assertions are off.

Refs COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lvaroqui
lvaroqui force-pushed the cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat branch from 2765419 to 5468bd4 Compare October 20, 2026 16:47
lvaroqui and others added 2 commits October 22, 2026 11:08
`init` installs its subscriber with `tracing::subscriber::set_global_default`, which, unlike
`tracing-subscriber`'s `init`, doesn't forward the records of the `log` crate to it: benchmarks
and dependencies logging with `log` exported nothing.

`init` now also installs `tracing-log`'s `LogTracer`, unless a `log` logger is already set. The
records go through the same `EnvFilter` as events, by their `log` target, and the bridge reads
their location from their normalized metadata, so they get the right `code.*` attributes. The
`log event` name they are given is already cleared.

`LogTracer` takes the max level of the subscriber, which the `log` macros check before anything
else. The `filter_fn` filters of the layer gave it no hint, which lifted it to `TRACE`: they now
hint `TRACE`, so that the `EnvFilter` hint is kept. `layer` stays free of global side effects,
and documents that a subscriber of your own needs a `LogTracer` to receive `log` records.

A test runs `init`, in a binary of its own since it sets process-wide state, against an
in-process OTLP/HTTP receiver, and checks the records it decodes. Like the export, it is only
built with `--cfg codspeed_telemetry`.

Refs COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first span of the process takes 20 to 30 µs longer than the next
ones to go through the subscriber, and the first event about 10 µs
longer. The first benchmark emitting them took that time, which delayed
its telemetry from the start of its run.

`init` now emits a span and an event once the subscriber is installed.
They carry a `codspeed_warmup` field, which the processors in front of
the exporters drop, and a target the filter always lets through at the
`error` level, which keeps the max level `log` takes from the filter.

Refs COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lvaroqui
lvaroqui force-pushed the cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat branch 3 times, most recently from ce1df9a to dfb0562 Compare October 23, 2026 15:59
@lvaroqui lvaroqui changed the title feat(divan): export benchmark tracing spans over OTLP behind an otel feature feat: export the telemetry of divan benchmarks over OTLP Oct 23, 2026
@lvaroqui
lvaroqui changed the base branch from main to feat/time-benchmarks-with-round-markers October 23, 2026 15:59
@lvaroqui
lvaroqui added this pull request to stack #202 October 23, 2026 16:00
@lvaroqui
lvaroqui marked this pull request as ready for review October 23, 2026 16:00
@greptile-apps

greptile-apps Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; neither previous finding remains outstanding.

Summary

Adds OTLP export for Divan benchmark spans and logs, with a separate telemetry build mode.

  • Divan benchmarks send their traces and logs to OTLP.
  • Cargo CodSpeed runs benchmarks once in telemetry mode.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Build["cargo codspeed build -m telemetry"] --> Bench["Divan benchmark"]
    Bench --> Spans["tracing spans"]
    Bench --> Logs["tracing events and log records"]
    Spans --> Traces["OTLP traces"]
    Logs --> Records["OTLP logs"]
    Traces --> Receiver["CODSPEED_OTEL_ENDPOINT"]
    Records --> Receiver
Loading

Reviews (2) · Last reviewed commit: "perf(codspeed): warm the telemetry up be..." · Reviewed by Greptile

Comment thread crates/codspeed/src/telemetry.rs Outdated
Comment thread crates/codspeed/src/telemetry.rs
@lvaroqui
lvaroqui force-pushed the cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat branch 2 times, most recently from 31f47a5 to 1e3f5f8 Compare October 26, 2026 08:59
@lvaroqui
lvaroqui requested a review from not-matthias October 26, 2026 11:58
@lvaroqui
lvaroqui force-pushed the cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat branch from 1e3f5f8 to 36e7664 Compare October 26, 2026 13:37
@lvaroqui
lvaroqui removed this pull request from stack #202 October 26, 2026 13:37
@lvaroqui
lvaroqui changed the base branch from feat/time-benchmarks-with-round-markers to main October 26, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant